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ABSTRACT 

Factors that influence the ef f activeness of program 
evaluation in higher education and eight recommendations for program 
evaluation are considered. Currently, program evaluation is 
characterized by negotiation because of three powerful concepts: the 
nature of multiple constructions and multiple realities; the 
influence of power distributions throughout higher education 
institutions; and the role of politics. The theory of multiple 
realities proposes that participants in organizations create 
realities about the organization that are based on multiply and often 
conflicting value systems. Evaluators must take each construction 
into account and recognize that no single reality exists for the 
organization. Power is not limited to those who have the funds of 
authority to commission evaluations, since others can give or 
withhold information or sway opinion. Since organizational politics 
is integral to the group's functioning, evaluations cannot avoid 
oolitics. implications and recommendations include: evaluators must 
understand that evaluation is not research but a teaching/learning 
process; evaluation is a continuous and divergent process; evaluation 
not only uncovers various reality constructions but creates 
realities; evaluation is an emergent process; and program evaluation 
is a collaborative process between equals. (SW) 
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I would like to begin today with several propositions which run 
counter to the mainstream in program evaluation. As typically conducted, 
the principles of good scientific inquiry usually obtain, and evaluation 
reports have more or less traditionally looked like research reports 
which went slightly askew. Elsewhere, I have contended that evaluation 
has moved from the era of comparison (for instance, to objectives) and 
description to the era of judgment, and finally to an era where the 
need is recognized for negotiation to be the hallmark characteristic. 

Negotiation comes to be the hallmark characteristic because of 
three powerful concepts: the nature of multiple constructions and 
multiple realities; the influence of power distributions throughoit institutions 
of higher education; and the role of politics in program evaluation. 
Those three concepts need to be defined, because they run counter to 
what most of us have been taught about social science inquiry, and 
because they will shape inevitably how effective program evaluation will 
be in the future. 

The concept of multiple realities grows out of a paradigm for 
inquiry which runs counter to the traditional or conventional paradigm, 
but it is not necessary to engage in long epistemological debates to see 
the relevance of the concept for proc;rs"n evaluation. Essentially, 
multiple realities captures the idea that participants in organizations 
create for themselves "realities""Constructions of what it is they are 
about, what it means, and how to get there from here (what Weick calls 
an "organizational, map"). Those realities are individual, although 
aspects of the realities are shared by members of the subgroup tc 
which the individual belongs. When large numbers of subgroups share 
elements of belief systems and constructions about the world they 
inhabit, they are said to participate in organizational cultures. The 
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"stories 11 which become part of the mythology (the reality) of the organizational 
culture become the organization's "saga". The important thing about 
multiple realities is that it flies in the face of conventional assumptions 
regarding just how many realities are "out there". This assumption 
demands that the evaluator confront the cold, hard truth that there is 
• no single reality which may be distilled from pieces which every organizational 
member might provide; that rather there are as many realities as there 
are groups or individuals who might construct them. Further, since 
those realities are based on multiple, pluralistic and often conflicting 
and competing value systems, each construction has to be taken into 
account in the process of evaluation. 

Mitroff has commented on the effects of multiple groups with multiple 
constructions and their role in organizational functioning earlier this 
year. Using an old term which described those who have a legitimate 
interest in program evaluation, he calls these groups and individuals 
"stakeholders of the organizational mind" (1985). 

The second concept is that of power and its distribution. In older 
forms of evaluation, our presumption was that power particularly 
evaluation power rested with those legitimately authorized to commission 
evaluation reports. This view has been termed "synoptic" for its 
narrow conception. Power is most assuredly not limited to those who 
have the money or formal authority to commission evaluations. It also 
rests with those who have information to give or withhold, with those 
who have legitimate and perhaps moral rights to the information we 
uncover, and with those whose careers and fortunes rest upon the 
judgments we make and the negotiations into which we enter. The 
uneven and somewhat fluid distribution of power throughout organizations 
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means that we cannot operate from a secure base ">f fiscal or programmatic 
legitimation. Others who have no money, but who have powar and can 
sway opinion, who can open doors or close avenues, can and must have 
their values sought, displayed and honored. 

Our older models of organizational functioning are simply no longer 
serviceable when laid alongside the realities of current forms of higher 
education. The implied linearity or verticality of bureaucratic of even 
collegial forms mask invisible connections which e~«st and which must be 
identified and taken account of in the evaluation process. 

Third, we are so caught'up in attempting to keep our data "clean" 
that we miss the simply observation that organizational politics is not 
"noise in the data", nor is it inescapable contamination or confounding. 
It is integral to and inextricably entertwined with the functioning of all 
human organizations. In a recent dissertation completed by a student, 
a Regent for the statewide institutions wistfully said, "I. wish just once 
we could accomplish a program review without all the politics getting in 
the way!" (Breier, 1985). Utterly lost on the individual was the political 
nature of institutions of higher education, the fact that information is 
power, the fact that while not all information is political, most information 
can be put to political ends, and thefact that what is intended is not 
necessarily what is implemented, and that in turn is not necessarily 
what is experienced by participants. (There must be hundreds of 
examples of this phenomenon which occur between Federal legislat ion 
and program enactment at the local level.) 

To the extent that we keep trying to avoid politics, or that we 
continue to believe that we could do better evaluations if we could do 
them without the politics, we are going to butt our heads against a wall 
again and again. 
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Implications for Effective Program Evaluation 

These three simple concepts dictate a whole new genre of program 
evaluation, tailored to recognition of new institutional forms, open to 
appropriate and inappropriate uses of power by stakeholders, conizant 
of the necessity of political give-rand-take, and sufficiently sophisticated 
to realize that recommendations which we as program evaluators see as 
pristine are nothing more than agendas for negotiation between stakeholding 
parties. The rules and roles of evaluators have changed, and the 
implications are potent. 

1. First and foremost, those engaged in program evaluation trust 
realize that while their activities fit comfortably within the definition of 
disciplined inquiry, evaluation is not research. Rather, it resembles 
much more a social-political process . As Cronbach warned us, "A 
theory of evaluation must be as much a theory of political interaction as 
it is a theory of how to determine .facts" (1980 p. 13). 

Higher education is a value-embedded proposition, and evaluation 
within it "inevitably serves some social agenda and becomes a tool of 
social advocacy 11 (Guba and Lincoln, 1985). Since evaluation acts as a 
tool to produce change, any changes proposed may be viewed by one 
group as highly desirably, and by still another as extremely unfortunate 
or even highly negative. As a result, those &ho hold stakes in the 
organization whether from inside or externally — have a right to 
voice thc?ir values and have those values honored in the proposition of 
any changes. To fail to "hear 11 or take account of the various value 
systems invariably robs some groups of their right to voice. 

2. Evaluation is a teaching/learning process . In the past, we 
have tended to act as though information and data flowed one way: 
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toward the evaluator. But informed judgments cannot be made by all 
stake holders until they know and understand the value positions held 
by other stakeholding groups. Therefore, one role of the evaluator is 
to move between groups — in a hermeneutic and dialectical process -- 
learning the value positions of each and in turn, teaching other groups 
about the various constructions of reality which are held. 

The implications are profound for the c ita collections process:, 
toth'for what is considered data, and with whom those data are to be 
shared. Clearly, this calls for a much larger conception of data (moving 
away from numbers, charts, projects), constructed to include explicit 
formulations of value and belief systems. The heavy reliance on numbers 
alone with cease to be a hallmark of program evaluation, while the 
negotiative and collaborative component assumes a larger share of the 
evaluation time spent. 

The role of the evalutor changes from that of design and display 
expert in the experimental mode to that of linchpin, teacher and learner 
in a series of circles, the purposes of which include understanding and 
negotiation of value positions. 

3*. Evaluation will move from being considered a discrete, time-bound, 
closed-ended and convergent eve n t to being seen as a continuous , recursiv e 
hermeneutic and divergent process : As a result, evaluations are never 
over, they are merely terminated at some time because of the lack of 
funds, lack of time, or the necessity to make a programmatic decision. 
As the effort moves between groups, understandings are enlarged, ne v 
questions arise, subtleties increase, and value positions are clarified. 
The process has the potential of Portnoy's onion: layer upon layer of 
orn^izational functioning may be uncovered in this recursive process. 

8 



6 



And the questions which might be profitably addressed grows exponentially. 

4. Evaluation is not merely a process which uncovers the various 
reality constructions which ara held by stakeholders, it also is a process which 
literally "creates" realities . According to the first concept, reality does 

r 

not exist in objective form, but rather is created by those persons who 
claim to know it. But such reality constructions are neither complete 
nor are they unchangeable. For that reason, as realities are shared 
between groups, individuals may alter their belief systems, may comprehend 
new elements which need to be added to the systems, or may simply 
become more informed and sophisticated about the positions of others. 
These changes literally create a new reality for some. 

The effect if you don't know what you're seeing is that the 
situation is undergoing change which has the potentia .o contaminate 
your evaluation findings. But contamination is a pathological word, 
implying that someone is messing up the evaluation design, when such 
changes are, in fact, exactly what was desired from the beginning. It 
is not a form of threat to the evaluation enterprise so much as it is a 
form of what Chris Argyris calls "double-loop learning" learning 
which fosters change and improved understanding. 

5. Evaluation must of necessity be an emergent process . Since 
the evaluator must begin with only some questions to be asnwered, or 
begin with the imperative of a decision's needing to be made, he or she 
has little idea of what is salient in the context, what values will impinge 
on decisions, what political influences will be brought to bear, or what 
the final agenda for negotiations must be, the design for the evaluation 
cannot be prespecified . Rather, it unfolds as various forms of data 
and constructions are uncovered and fed into the process. This is an 
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<*jbrupt change from the previous eras where evaluation designs were as 
formally specified as* were those for experimental research. 

The disadvantage of this implication is that evaluation design is 
not subject to closure (until after the evaluation is completed, when all 
the steps may be specified retrospectively). Ine advantage is that 
open-ended designs allow the full interplay of political and power forces 
which characterize institutions to come to the fore--without the necessity 
of believing that they are contaminating the evaluation effort. In fact, 
open-ended designs encourage the expression of power and influence 
(politics) since that expression is the visible characteristic of value 
positions of stakeholders. 

7. Program evaluation in institutions of higher education becomes 
less a set of judgments rendered by outside experts, and more of 
a collaborative, transactional and negotiated process engaged in between equals 
Different value postures can lead to quite different judgments even on 
the basis of the same evidence, so negotiation among different value 
holders is mandated if the evaluation effort is to be viewed ultimately as 
fair and balanced (notice that I didn't say "objective 1 '). Recommendations 
which are derived from the value position of the evaluator alone, or 
from the contractor and evaluator in consort, ignore the value positions 
of legitimate stakeholders, who are exploited by such a process. 

The appropriate outcome for such a form of evaluation is not a set 
of recommendation? derived by the evaluator, or his administrative 
client. It is rather an agenda for negotiation between interested parties 
which faithfully depicts and incorporates the value positions the value 
positions of each group of stakeholders. 
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Clearly, the openness of evaluation findings to negotiation inv ; tes 
the full play of political forces. However, it invites that full play in 
ways in which guarantee public disclosure and which discourage under- 
the-table and behind-closed-doors power alignments. The commitment to 
bargain over a set of agenda items demands that all parties present 
their items for consideration, and encourages the open and public 
debate over values which ought to characterize democratic or pluralistic 
organizations. In this situation, to fail to come forward with recommendations 
for negotiations is self-destructive. Thus, negotiating openly forces 
political interests into the arena of other competing social and political 
interests in ways which insure more compromise and fewer secret coalitions. 

8. Finally, the agenda for negotiation takes on a very different 
format from the earlier technical report which characterized first- and 
second-generation evaluation. The agenda is often presented in the 
form of a case study, with both quantitative and qualitative data setting 
out the particularly of this context and this decision arena. Thus the 
negotiating agenda remains embedded in the institutions own particular 
history and contemporary situation. The particularly displayed this 
way in case study format provide a basis for discussion among 
and between stakeholders, as well as recreatinq the multiple realities 
which co-exist in the institution. 

Conclusion 

The set of eight implications above are stated simply and on the 
face of it, seem intuitive. They do not represent most of the practice 
of program evaluation in higher education, however, which has typically 
been conducted on the model o f experimental research design and carri^" 
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out with M hired guns M --consultants from outside the institutional context 
whose first loyalties are to their clients (rather than to legitimate 
stakeholders) and to their canons for objectivity and impersonality 
(rathe than to fairness, balance, and political realities). 

New forms of program evaluation in higher education call for new 
roles, roles which hav* been filled more from the ranks of senior and 
prestigious faculty than from and by professional evaluators. The new 
program evaluator in higher education will be less a statistical expert 
and outside or external consultant than she or he will be a collaborator, 
a teacher and learner simultaneously, a reality shaper, and a mediator, 
negotiator and change agent. Rather than providing data arrays to 
institutions which afte- all, are readily available from the institution's 
own institutional research division the evaluator will be skillful at 
locating and presenting the belief systems of stakeholders; will be 
adroit at mediating between political positions on campus and off; will 
not leave the scene at the delivery of the final report, but rather will 
move into a negotiating mode as senior arbiter in achieving compromise; 
and will be one of many forces channeling programmatic change on 
campuses. The model for these roles is less that of technician than 
that of anthropological fieldworker and labor relations arbitration expert. 

Effective program evaluation in higher education is finally aided 
more by a recognition of the forces which shape the enterprise than by 
intense methodological and technical preparation. The three roost potent 
forces are the pluralistic value positions, the distribution of power 
among and between professionals and pre-professionals, and the embedded 
ness of politics in everyday life. A model for evaluation, which not only 
takes into account these forces, but wncih uses them to enlighted and 



9 

ERIC 



12 



10 



enlarge f he understandings of all groups, has the greater power to 
undertake and ultimately sustain desired change. 
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